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ABSTRACT 

Some  high-performance  imaging  systems  generate  a  curved  focal  surface  and  so  are  incompatible  with 
focal  plane  arrays  fabricated  by  conventional  silicon  processing.  One  example  is  a  monocentric  lens, 
which  forms  a  wide  field-of-view  high-resolution  spherical  image  with  a  radius  equal  to  the  focal  length. 

Optical  fiber  bundles  have  been  used  to  couple  between  this  focal  surface  and  planar  image  sensors. 

However,  such  fiber-coupled  imaging  systems  suffer  from  artifacts  due  to  image  sampling  and  incoherent 
light  transfer  by  the  fiber  bundle  as  well  as  resampling  by  the  focal  plane,  resulting  in  a  fixed  obscuration 
pattern.  Here,  we  describe  digital  image  processing  techniques  to  improve  image  quality  in  a  compact 
126°  field-of-view,  30  megapixel  panoramic  imager,  where  a  12  mm  focal  length  F?  1.35  lens  made  of  concentric 
glass  surfaces  forms  a  spherical  image  surface,  which  is  fiber-coupled  to  six  discrete  CMOS  focal  planes. We 
characterize  the  locally  space-variant  system  impulse  response  at  various  stages:  monocentric  lens  image  formation 
onto  the  2.5  ?m  pitch  fiber  bundle,  image  transfer  by  the  fiber  bundle,  and  sensing  by  a  1.75  ?m  pitch  backside 
illuminated  color  focal  plane.  We  demonstrate  methods  to  mitigate  moire  artifacts  and  local  obscuration,  correct  for 
sphere  to  plane  mapping  distortion  and  vignetting,  and  stitch  together  the  image  data  from  discrete  sensors  into  a 
single  panorama.  We  compare  processed  images  from  the  prototype  to  those  taken  with  a  10x  larger  commercial 
camera  with  comparable  field-of-view  and  light  collection. 
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Some  high-performance  imaging  systems  generate  a  curved  focal  surface  and  so  are  incompatible  with 
focal  plane  arrays  fabricated  by  conventional  silicon  processing.  One  example  is  a  monocentric  lens, 
which  forms  a  wide  field-of-view  high-resolution  spherical  image  with  a  radius  equal  to  the  focal  length. 
Optical  fiber  bundles  have  been  used  to  couple  between  this  focal  surface  and  planar  image  sensors. 
However,  such  fiber-coupled  imaging  systems  suffer  from  artifacts  due  to  image  sampling  and  incoherent 
light  transfer  by  the  fiber  bundle  as  well  as  resampling  by  the  focal  plane,  resulting  in  a  fixed  obscuration 
pattern.  Here,  we  describe  digital  image  processing  techniques  to  improve  image  quality  in  a  compact 
126°  field-of-view,  30  megapixel  panoramic  imager,  where  a  12  mm  focal  length  F /1.35  lens  made  of  con¬ 
centric  glass  surfaces  forms  a  spherical  image  surface,  which  is  fiber-coupled  to  six  discrete  CMOS  focal 
planes.  We  characterize  the  locally  space-variant  system  impulse  response  at  various  stages:  monocentric 
lens  image  formation  onto  the  2.5  pm  pitch  fiber  bundle,  image  transfer  by  the  fiber  bundle,  and  sensing 
by  a  1.75  pm  pitch  backside  illuminated  color  focal  plane.  We  demonstrate  methods  to  mitigate  moire 
artifacts  and  local  obscuration,  correct  for  sphere  to  plane  mapping  distortion  and  vignetting,  and  stitch 
together  the  image  data  from  discrete  sensors  into  a  single  panorama.  We  compare  processed  images 
from  the  prototype  to  those  taken  with  a  lOx  larger  commercial  camera  with  comparable  field-of-view 
and  light  collection.  ©  2015  Optical  Society  of  America 

OCIS  codes:  (110.0110)  Imaging  systems;  (100.0100)  Image  processing;  (110.1758)  Computational 
imaging. 

http://dx.doi.org/10.1364/AO.54.001124 


1.  Introduction:  Traditional  Imagers  and  Fiber-Coupled 
Imagers 

Traditional  imaging  systems  form  images  onto  flat 
image  sensors,  a  difficult  design  constraint  for 
wide-angle  imaging  in  which  off-axis  aberrations 
dominate.  A  monocentric  lens,  where  all  optical  sur¬ 
faces  are  spherical  and  share  a  common  center  of  cur¬ 
vature,  images  without  coma  or  astigmatic  aberration 
and  significantly  reduces  imager  volume  (Fig.  1)  [1,2]. 
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This  comes  at  a  cost  of  forming  a  spherical  image 
surface,  which  is  incompatible  with  conventional 
image  sensor  fabrication  technology  [3],  Fiber  optic 
bundles,  a  dense  array  of  high  index  optical  fibers 
with  thin  low  index  cladding,  can  be  used  to  incoher¬ 
ently  transfer  optical  power  between  an  input  and 
output  surface  [4,5]. 

This  technique  was  used  to  couple  the  spherical 
image  surface  from  a  60°  field-of-view  monocentric 
lens  to  one  or  more  221  kpixel  CCD  sensors  with 
23  pm  pitch  pixels  [6].  More  recently,  the  same  basic 
structure  has  been  applied  to  double  the  lens  field-of- 
view  and  couple  the  image  to  high-resolution  CMOS 
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Fig.  1.  (a)  Conventional  lens  forms  an  aberrated  planar  image  (the  prototype  is  inset  to  show  relative  size).  The  Zebase  design  F_005  has 
been  reoptimized  to  achieve  f  =  12  mm,  F / 1.7,  and  120°  field-of-view.  (b)  Prototype:  a  monocentric  lens  forms  low-aberration  spherical 
images  but  requires  fiber  bundles  to  relay  the  image  to  flat  sensors. 


sensors  [7,8] .  The  lens,  fiber  bundle,  and  fiber- 
coupled  sensor  are  shown  in  Fig.  2,  and  a  more  de¬ 
tailed  view  of  the  structure  of  the  fiber  bundle  and 
sensor  is  shown  in  Fig.  3.  Signal  acquisition  in  such 
fiber-coupled  imagers  results  from  a  cascade  of  opti¬ 
cal  processes.  A  continuous  image  is  formed  by  the 
lens  on  the  fiber  bundle  surface,  where  the  local  sig¬ 
nal  coupling  may  be  strongly  space  variant  according 
to  the  quasi-periodic  fiber  bundle  structure.  This  sig¬ 
nal  is  transferred  through  the  multimode  fiber  core, 
where  it  loses  all  spatial  coherence  then  is  emitted 
from  the  output  face  of  the  fiber.  In  general,  there 
can  be  a  small  free  space  gap  (an  air  space  or  thick¬ 
ness  of  isotropic  adhesive)  between  the  rear  face  of 
the  fiber  and  the  focal  plane,  and  the  signal  emission 
into  this  region  can  depend  sensitively  on  launch  con¬ 
ditions  and  wavelength.  Finally,  this  signal  is  cap¬ 
tured  by  the  regular  sampling  grid  of  the  focal 
plane  sensor.  If  the  fiber  pitch  is  much  smaller  than 
the  sensor  pixels,  this  image  acquisition  “cascade” 
might  be  neglected.  To  achieve  the  highest  possible 
spatial  resolution,  however,  both  fiber  bundle  and 
pixel  pitch  need  to  be  minimized.  Understanding 
each  step  of  the  image  acquisition  process  can  enable 
a  broader  class  of  computational  image  processing 
techniques. 


Fig.  2.  (a)  Monocentric  lens  (MC)  comprised  of  a  symmetric  two- 
glass  achromat  (top)  and  meniscus  lens  (bottom),  (b)  Fiber  bundle 
relays  a  business  card  logo  from  its  flat  to  its  curved  surface, 
(c)  Fiber  bundle  coupled  to  a  sensor  using  optical  epoxy. 


Here,  we  present  the  first  (to  our  knowledge) 
systematic  characterization  of  the  spatially  variant 
impulse  response  at  various  stages  within  a  fiber- 
coupled  imager  (an  imaging  system  that  uses  fiber 
bundle  image  relay).  We  use  a  fiber-coupled  mono¬ 
centric  lens  camera  prototype  to  investigate  compu¬ 
tational  imaging  techniques  applicable  to  any 
imaging  system  that  uses  fiber  bundles  to  relay  or 
field-flatten  images  onto  planar  image  sensors  as 
well  as  system-specific  issues  related  to  combining 
multiple  sensors  to  form  visually  seamless  panora¬ 
mas.  Section  2  is  devoted  to  an  in-depth  investigation 
of  the  local  image  transfer  characteristics  of  each 
component  within  the  prototype.  Section  3  describes 
the  methods  we  employed  to  globally  remove  image 
artifacts  due  to  fiber  optic  bundle  image  transfer  and 
system  components.  Section  4  shows  the  results  of 
mapping  the  spherical  image  space  to  a  planar  space 
and  how  to  form  panoramas  by  aligning  and  stitching 
contiguous  sensor  data.  Section  5  explains  the  exper¬ 
imental  setup  under  which  data  is  collected  for  cam¬ 
era  calibration  and  performance  measurements. 
Section  6  discusses  our  experimental  results  from 
the  prototype  imager  and  compares  the  performance 
to  a  much  larger  conventional  digital  SLR  “bench¬ 
mark”  camera  (Fig.  4). 

2.  Component  and  System  Point  Spread  Function 

We  characterized  our  custom  monocentric  lens  fabri¬ 
cated  by  Optimax  (focal  length  f  =  12  mm,  F /  = 
1.35)  [Fig.  2(a)]  [9]  and  compared  it  to  a  high- 
resolution  Nikon  LU  Plan  APO  150X  microscope  ob¬ 
jective  (f  =  1.33  mm,  numerical  aperture  NA  =  0.9). 
Both  of  these  objectives  were  used  with  the  following 
devices  under  test  (DUTs)  to  sample  the  image 
formed:  a  24AS  Schott  fiber  bundle  [Figs.  2(b)  and 
3(a)]  with  2.5  pm  fiber  pitch  [10,11],  an  Omnivision 
OV5653  backside  illuminated  5  megapixel  color 
CMOS  sensor  [Fig.  3(b)]  with  1.75  pm  pixel  pitch 
and  image  area  of  4592  pm  wide  by  3423  pm  high 
[12],  and  a  fiber-coupled  sensor  formed  by  bonding 
the  fiber  bundle  to  the  sensor  using  UV-cured 
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Fig.  3.  (a)  Fiber  bundle  structure  and  defects  lead  to  image  artifacts.  Fiber  absorbers  between  every  other  2.5  |im  fiber  intersection 
prevent  crosstalk,  (b)  OVT5653  sensor  is  used  to  sample  the  image  exiting  the  fiber  bundle.  The  two  magnified  inset  micrographs 
are  of  the  same  scale. 


Fig.  4.  (a)  Side-by-side  size  comparison  of  the  prototype  with  anF/4  commercial  camera  of  similar  field-of-view.  (b)F/1.35,  30  megapixel, 
126°  field-of-view  fiber-coupled  monocentric  lens  imager  prototype. 


Norland  optical  adhesive  NOA72  [Fig.  2(c)].  The 
internal  structure  of  the  fiber  bundle  is  shown  in 
Fig.  3(a)  and  leads  to  artifacts  in  the  relayed  imagery. 
A  schematic  and  image  of  the  cross  section  of  the 
fiber-coupled  sensor  is  shown  in  Fig.  5.  The  fiber 
bundle’s  input  face  has  been  cut  and  polished  with 
a  12  mm  radius  of  curvature  to  mate  with  the  menis¬ 
cus  lens.  A  pedestal  has  been  cut  out  of  its  exit  face  to 
mate  with  an  image  sensor  whose  cover  glass  has 
been  removed.  The  schematic  illustrates  that  the 
light  will  pass  through  the  lens,  fiber  bundle,  epoxy 
layer,  lens  array,  and  Bayer  filter  before  it  is  collected 
by  the  image  sensor.  The  epoxy  layer  must  be  kept  to 
a  minimum  thickness  in  order  to  prevent  the  diver¬ 
gent  light  exiting  the  fiber  bundle  to  defocus  before  it 
is  collected  by  the  image  sensor.  The  fiber  bundle’s 
polished  surface  flatness  is  on  the  order  of  a  wave¬ 
length,  and  the  OV5653  surface  flatness  variations 
are  approximately  1.5  pm,  as  measured  with  the 
Veeco  NT1100  Optical  Profiling  System.  A  cross  sec¬ 
tion  of  one  of  the  fiber-coupled  sensors  confirmed  a 
2  pm  adhesive  thickness  over  the  sensor’s  1.6  pm 
thick  lenslet  and  color  filter.  Extremely  wide-angle, 
spatially  and  temporally  incoherent  light  emission 


through  these  surface  and  interface  conditions  do 
not  generate  high-contrast,  submicron  spatial  pat¬ 
terns;  therefore,  negligible  interference  effects  are 
detected  by  the  sensor. 

To  measure  the  impulse  response  of  the  DUTs,  we 
built  an  optical  setup  to  simulate  a  moving  infinitely 
distant  point  source  using  a  white  LED  light  behind 
a  small  (3  pm)  pinhole  aperture  at  the  focal  plane  of  a 
long  focal  length  Fourier-transform  lens  (f  = 
360  mm,  F /  =  5.14,  and  diameter  0  =  76  mm) 
(Fig.  6).  The  pinhole  and  LED  are  both  mounted 
on  a  precision  translation  stage  and  shifted  in  small 
lateral  steps  to  characterize  the  highly  localized  spa¬ 
tial  variations  in  the  impulse  response  of  the  DUTs. 
The  total  translation  corresponded  to  a  25  pm  dis¬ 
placement  at  the  DUT  and  was  discretized  into 
200  steps.  For  each  step,  an  image  was  captured, 
and  all  images  were  then  used  to  form  the  3D  data 
cubes  shown  in  Fig.  7  and  the  data  projections  in 
Fig.  8.  To  form  the  data  cube,  the  image  frames  in 
x,  y  are  stacked  up  along  z.  The  x,  y  coordinates  of 
the  data  cube  are  in  the  pixel  domain  of  the  sensor. 
The  z  axis  contains  the  individual  photographs  cap¬ 
tured  by  the  sensor  for  each  step  of  the  point  source 
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Fig.  5.  (a)  Schematic  and  (b)  cross  section  showing  the  internal 
structure  and  scale  of  the  fiber-coupled  sensor.  The  fiber  bundle 
relays  light  incident  at  its  surface  to  the  CMOS  image  sensor 
though  an  epoxy  bonding  layer,  lens  array,  and  Bayer  filter. 


translation.  Projections  onto  each  coordinate  axis 
plane  are  used  to  help  interpret  the  data.  The  projec¬ 
tion  onto  the  bottom  plane  shows  the  integrated  re¬ 
sponse  of  the  sensor  for  all  frames  captured  during 
translation.  Projections  onto  the  two  side  planes 
show  the  behavior  due  to  the  translation,  where  each 
frame  is  integrated  in  one  dimension  to  form  these 
side  projections.  We  repeated  this  process  for  each 
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Fig.  6.  Schematic  of  the  impulse  response  characterization  setup 
where  an  LED,  pinhole,  and  collimation  lens  if  =  360  mm, 
F/  =  5.14,  0  =  76  mm)  form  a  collimated  point  source.  Setup  in 
(a)  uses  a  high-resolution  microscope  objective  (f  =  1.33  mm, 
NA  =  0.9)  to  characterize  the  monocentric  lens  in  setup  (b).  Light 
emitted  by  the  LED  is  focused  onto  a  DUT.  Translation  stage  is 
used  to  move  the  point  source  so  that  the  focused  spot  moves 
25  pm  laterally  across  the  surface  of  the  DUT.  This  characteriza¬ 
tion  is  done  to  measure  the  local  spatial  variance  of  the  impulse 
response  for  each  DUT.  The  DUTs  are  a  fiber  bundle,  sensor, 
and  fiber-coupled  sensor  (see  Fig.  8). 


DUT  using  the  microscope  objective  and  monocentric 
lens.  Figure  8  presents  the  data  collected  but  only 
shows  two  of  the  three  projections  of  the  data  cube, 
namely,  the  x-y  pixel-pixel  sensor  domain  and  the 
x-z  pixel-distance  domain. 

A  Nikon  LU  Plan  APO  150X  microscope  objective 
(1.6  pm  spot  size)  was  used  to  illuminate  each  DUT  to 
produce  the  data  in  the  top  row  of  Fig.  8,  while  a  cus¬ 
tom  monocentric  lens  (2.6  pm  spot  size)  illuminated 
each  DUT  producing  data  shown  on  the  bottom  row 
of  Fig.  8  [see  schematics  in  Figs.  6(a)  and  6(b),  respec¬ 
tively].  The  spot  size  was  defined  by  the  e~l  intensity 
level.  For  comparison,  Figs.  8(a)  and  8(e)  show  the 
ideal  impulse  response  of  the  microscope  objective 
and  monocentric  lens,  respectively  (measured  using 
a  1600  x  1200  resolution,  4.4  pixel  pitch  Keyence 
VHX-1000  microscope  with  a  20x  optical  magnifica¬ 
tion  lens  VH-Z100UW  of  NA<0.48  and  exposure  time 
of  l/60th  of  a  second).  Both  follow  the  image  of  the 
translated  point  source  smoothly  and  continuously, 
meaning  these  devices  are  spatially  invariant. 
The  width  of  the  line  corresponds  to  the  spot  size 
produced  by  each  lens,  namely,  1.6  and  2.6  pm. 
Figures  8(b)  and  8(f)  show  the  same  impulse 
response  measured  using  an  Omnivision  5653  sensor 
without  any  optical  magnification.  The  Bayer- 
filtered  1.75  pm  pixels  cause  the  spatial  quantization 
in  image  sensor  space.  The  difference  in  the  slope  in 
Fig.  8(f)  when  compared  to  the  rest  of  the  figure  is 
due  to  the  index-matching  fluid-filled  gap  between 
the  planar  sensor  and  spherical  image  surface. 
Figures  8(c)  and  8(g)  show  the  same  impulse 
response  as  Figs.  8(a)  and  8(e)  after  being  relayed 
by  the  fiber  bundle  and  measured  with  the  Keyence 
microscope.  These  images  exhibit  large  variation  in 
transmission  depending  on  the  location  of  the 
impulse  relative  to  the  fiber  structure,  but  this  is  pri¬ 
marily  due  to  the  light’s  higher  divergence  angle  at 
the  fiber’s  exit  face  (when  incident  at  the  noncore  re¬ 
gions),  which  cannot  be  captured  because  it  is  beyond 
the  microscope’s  numerical  aperture.  Figures  8(d) 
and  8(h)  show  data  collected  by  a  image  sensor 
coupled  directly  to  the  exit  face  of  the  fiber  bundle. 
There  is  some  image  spread  but  more  uniform  energy 
transmission.  A  similar  evolution  of  the  impulse 
response  of  the  DUTs  in  Figs.  8(e)-8(h)  is  also  seen 
in  the  corresponding  data  in  Fig.  7. 

Figures  8(c),  8(d),  8(g),  and  8(h)  show  the  same 
measurements  as  in  Figs.  8(a),  8(b),  8(e),  and  8(f), 
except  a  fiber  bundle  is  introduced  into  the  system 
to  relay  the  image.  The  fiber  bundles  used  in  our  sys¬ 
tem  were  made  by  Schott  using  24AS  glass,  with  a 
five-sided  core  structure  and  a  2.5  pm  pitch  (for  more 
information,  see  [10,11])-  Figures  8(c)  and  8(g)  show 
that  the  fiber  bundle  impulse  response  is  extremely 
spatially  variant  on  a  fine  spatial  scale  (on  the  order 
of  the  pixels),  allowing  a  large  signal  to  pass  when 
light  is  incident  on  one  of  the  fiber  cores.  Note  here 
that  the  fiber  bundle  does  transmit  light  incident  on 
noncore  regions,  but  that  the  Keyence  microscope’s 
numerical  aperture  (NA<0.48)  does  not  collect  the 
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Fig.  7.  Data  cube  representation  of  the  impulse  response  of  various  DUTs  corresponding  to  the  bottom  row  of  Fig.  8  collected  using  the 
setup  shown  in  Fig.  6(b).  The  x-y  axes  correspond  to  the  image  sensor  space,  while  the  z  axis  is  formed  by  stacking  up  200  images  taken 
during  the  25  pm  translation  of  a  point  source. 


highly  divergent  emitted  light.  However,  this  diver¬ 
gent  light  is  detected  by  a  CMOS  sensor  placed  in 
contact  with  the  fiber  bundle.  Spatially  variant  im¬ 
pulse  response  behavior  severely  limits  the  image 
processing  that  can  be  used  on  such  a  system.  Fortu¬ 
nately,  Figs.  8(d)  and  8(h)  show  that  a  fiber  bundle 
that  has  been  bonded  to  an  image  sensor  exhibits 
considerably  less  spatial  variance  than  the  fiber  bun¬ 
dle  alone  [see  Figs.  8(c)  and  8(g)] .  This  study  shows 
the  evolution  of  the  impulse  response  behavior  at  the 
various  stages  within  the  fiber-coupled  monocentric 
lens  imager.  It  reveals  that,  although  the  impulse  re¬ 
sponse  is  highly  spatially  variant  within  the  system, 
the  end-to-end  system  performance  is  in  general  not 
highly  spatially  variant.  This  enables  us  to  use  image 
processing  techniques  such  as  deconvolution  [13,14] 
in  order  to  deblur  images  or  extended  depth-of-focus 
in  order  to  improve  image  quality  [15,16], 

3.  Global  Artifact  Removal  Calibration  of 
the  Fiber-Coupled  Sensor  and  Prototype 

The  fiber  bundle’s  internal  structure  produces  an  oc¬ 
clusion  pattern  that  is  superimposed  on  the  images  it 


transfers.  Strong  sampling  effects  unique  to  fiber- 
coupled  imagers  are  produced  when  the  irregular 
fiber  lattice  and  periodic  pixel  lattice  are  of  compa¬ 
rable  pitch,  which  leads  to  misalignment  between 
the  lattice  sites,  causing  moire  patterns.  There  have 
been  efforts  to  remove  the  occlusion  effects  related  to 
fiber  bundle  image  relay  using  techniques  such  as 
spatial  filtering  [17],  Gaussian  smoothing  [18],  or  a 
Bayesian  framework  [19] .  However,  all  of  these  tech¬ 
niques  reduce  image  resolution.  The  objective  here 
is  similar  to  work  presented  in  [18,19]  with  more 
general  obscuration/occlusion  removal  techniques 
documented  in  [20-24]. 

Modern  CMOS  focal  planes  achieve  high  apparent 
uniformity  using  flat-field  calibration  and  analog 
processing  that  substantially  masks  unavoidable 
point  defects,  local  variations  in  sensitivity,  and  fixed 
pattern  noise.  Flat-fielding  has  been  used  to  correct 
fiber  relayed  images  [25]  and  spectra  [26].  We  ap¬ 
plied  a  similar  process  to  calibrate  the  fiber-coupled 
sensor  by  experimentally  acquiring  a  flat-field  obscu¬ 
ration  map  for  a  rigidly  attached  fiber-coupled  sensor 
or  fiber-coupled  monocentric  lens  imager.  This  map 
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Fiber  Bundle  Relay 


No  Fiber  Bundle  Relay 


Keyence  Microscope 


Omnivision  Sensor  Keyence  Microscope 


Omnivision  Sensor 


Fig.  8.  Impulse  characterization  of  system  components  where  each  subfigure  contains  (from  top  to  bottom)  the  integration  of  the  200 
images  captured  during  a  point  source  scan,  a  projection  of  the  data  onto  translation-pixel  space  (see  Fig.  7),  and  an  image  of  the  setup. 
These  plots  show  the  focus  and  linearity  for  each  DUT.  (a)-(d)  and  (eHh)  were  captured  using  the  setups  in  Figs.  6(a)  and  6(b),  respectively. 
The  difference  in  the  slope  of  the  data  presented  in  6(f)  is  due  to  the  planar  sensor  being  placed  at  the  spherical  image  surface,  which 
introduces  an  index-matching  fluid-filled  gap  at  the  edges  where  light  propagates  before  reaching  the  sensor.  Although  the  intermediate 
response  is  spatially  variant  6(g),  invariance  is  recovered  by  the  complete  prototype  6(h).  This  extends  the  applicable  image  processing 
techniques  that  can  be  used. 


quantifies  artifacts  that  affect  the  system  across  the 
overall  image  space,  whether  introduced  by  the  fiber 
structure  or  coupling  among  lens,  fiber  bundle,  and 
sensor.  It  also  allows  correction  of  field  nonuniform¬ 
ity  due  to  radial  vignetting  due  to  the  angle  of  inci¬ 
dent  light  on  the  fiber’s  input  face  as  well  as  lens 
vignetting.  The  flat-field  obscuration  map  is  essen¬ 
tially  an  image  of  an  evenly  illuminated  white  planar 
target  object  acquired  by  the  imaging  system.  The 


target  should  provide  uniform  illumination  free  of 
any  structure  and  does  not  require  the  target  be  in 
focus.  The  dark-field  image  is  used  to  account  for 
the  dark  current.  These  artifacts  due  to  the  lens, 
fiber  bundle,  adhesive,  and  sensor  can  have  higher 
order  dependence  on  system  temperature  (due  to 
differential  thermal  expansion)  or  sensor  gain  and 
exposure  settings  (due  to  nonlinear  sensor  response), 
leading  to  the  use  of  either  multiple  calibration 
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Table  1.  Statistics  Taken  over  the  Same  White  Part  of  the  Images  in 
Fig.  10  before  and  after  Occlusion  Removal  Show  the  Process 
Significantly  Improves  Image  Uniformity 


Uncalibrated  Image 
[Fig.  10(b)] 

Calibrated  Image 
[Fig.  10(c)] 

Color  Channel 

Mean 

Std.  Dev. 

Mean 

Std.  Dev. 

Red 

178.35 

20.59 

219.27 

9.78 

Green  1 

175.34 

23.88 

215.24 

7.69 

Green2 

174.04 

24.52 

215.16 

7.71 

Blue 

159.86 

17.73 

214.48 

6.87 

images  or  a  parametrized  response  function.  This 
idea  was  presented  in  [27,28],  which  used  multiple 
calibration  images  to  deal  with  nonlinear  intensity 
response.  This  is  a  topic  of  ongoing  research,  but  pre¬ 
liminary  results  indicate  the  approach  is  robust  to 
moderate  temperature  variations  (20C)  and  a  range 
of  exposures. 

The  flat-field  calibration  images  show  that  the 
finely  structured  global  attenuation  moire  pattern 
that  overlaid  the  image  is  deterministic  (Fig.  9). 
The  most  attenuated  values  fall  to  a  level  of 
about  65%  of  the  peak  intensity  on-axis  [Fig.  9(a)]. 


Figure  9(b)  shows  the  vignetting  effects  that  lead 
to  the  significant  attenuation  (see  left  side  of  the  cal¬ 
ibration  image)  and  shifting  of  the  histogram.  Even 
in  this  case,  the  signal  level  is  above  10%  of  the  peak 
intensity.  In  both  cases,  the  distribution  that  cen¬ 
tered  below  the  10%  value  corresponds  to  a  region 
of  the  sensor,  which  is  not  coupled  to  the  fiber  bundle 
(dark  regions  at  the  edge  of  the  calibration  images) 
and  can  be  ignored  since  it  does  not  contain  image 
data.  For  these  reasons,  the  flat-field  calibration  im¬ 
age  can  be  used  to  amplify  the  signal  of  subsequent 
images.  Figure  10  shows  the  calibration  image 
(a)  taken  of  a  flat  white  scene,  an  image  taken  of 
an  indoor  laboratory  scene  (b),  and  the  corrected 
image  (c).  The  correction  was  made  by  element-wise 
division,  which  compensated  for  attenuation  of  the 
signal.  The  normalization  factor  for  each  pixel  was 
obtained  from  the  corresponding  pixel  within  the  cal¬ 
ibration  image.  In  the  most  attenuated  regions  of  the 
image,  this  process  was  still  extremely  effective. 
Noise  in  the  flat-field  calibration  image  is  amplified 
and  can  be  significant  in  darker  regions  of  the  image, 
which  require  strong  amplification.  Although  the  ef¬ 
fects  are  not  significant  in  our  results,  they  can  be 


Fig.  9.  Two  typical  flat-field  calibration  images  (left)  and  their  histograms  (right)  captured  using  the  fiber  bundle  relayed  imager.  The 
lowest  intensities  transmitted  by  the  bundle  are  above  175  (65%)  (a)  on-axis  and  25  ( 10%)  (b)  off-axis  on  an  8-bit  scale.  Off-axis  intensity 
transmission  suffers  from  vignetting.  The  intensity  values  below  10%  correspond  to  a  region  of  the  sensor  where  there  is  no  fiber  bundle 
image  relay  and  can,  therefore,  be  ignored.  The  large  scale  structure  in  (b)  is  due  to  delamination  in  the  optical  adhesive;  yet,  this 
calibration  is  still  effective. 
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(a) 


(c) 

Fig.  10.  Calibration  image  of  a  flat  white  scene  (a)  shows  artifacts  due  to  broken  fibers,  scratches,  and  other  imperfections  as  well  as  the 
fiber  bundle’s  internal  structure,  all  of  which  lead  to  obscuration  and  color  artifacts  (b ).  These  artifacts  inherent  to  fiber  bundle  imaging  are 
corrected  using  a  calibration  image  (c).  The  center  white  region  in  the  magnified  views  of  (b)  and  (c)  are  used  to  quantify  the  calibration’s 
effectiveness  (see  Table  1). 


reduced  by  averaging  multiple  dark-field  and  flat- 
field  images,  respectively.  Areas  that  may  not  be 
remedied  by  this  process,  regions  of  the  calibration 
image  where  the  signal  was  below  a  certain  thresh¬ 
old,  could  have  been  further  corrected  by  interpolat¬ 
ing  from  nearby  regions.  Table  1  quantifies  the 
improvement  in  uniformity  over  a  uniform  white 
area  in  Figs.  10(b)  and  10(c).  It  was  shown  in  [29]  that 
the  flat-fielded  images  still  contain  remnants  of 
moire  “beat  patterns,”  which  can  be  used  to  measure 
fiber  bundle  distortion.  This  residual  variation  is 
evident  in  the  calibrated  image  in  Fig.  10(c)  and 
its  corresponding  statistics  in  Table  1. 

4.  Spherical  to  Planar  Mapping,  Image  Alignment,  and 
Stitching  Together  Image  Seams 

Even  with  an  ideal  lens,  there  is  an  intrinsic  distor¬ 
tion  associated  with  mapping  a  portion  of  a  spherical 


image  onto  a  flat  image  sensor,  and  the  distortion  is 
increasingly  apparent  as  the  field-of-view  recorded 
with  a  single  sensor  increases.  This  distortion  can 
be  corrected  using  one  of  several  general  mathemati¬ 
cal  transformations  [30].  The  mapping  of  a  direct 
spherical  projection  onto  a  plane  is  given  by  s  = 
2 r  -  Rr/^/R2  -  r2,  where  the  radial  distance  r  is 
transformed  into  a  scaled  form  s.  R  is  the  radial  dis¬ 
tance  from  the  point  where  the  spherical  surface  tan¬ 
gent  plane  is  orthogonal  to  the  sensor  plane.  Here 
Cartesian  coordinates  are  transformed  into  azi¬ 
muth-elevation  global  coordinates.  Each  sensor  is 
mapped  radially  in  this  way  to  correct  for  radial  pin¬ 
cushion  distortion  and  to  aid  in  stitching  individual 
sensor  data  to  form  panoramas. 

Assembly  of  the  imager  involves  affixing  the 
fiber-coupled  sensor  modules  to  the  lens  system.  In 
addition  to  cutting  the  input  and  output  faces  of 
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Fig.  11.  Image  processing  data  flow. 


the  fiber  bundles,  the  sides  are  also  cut  such  that 
multiple  fiber-coupled  sensors  can  fit  together  closely 
to  form  a  contiguous  sphere  of  the  same  radius  as  the 
meniscus  lens  so  the  two  can  interface.  Physical 
alignment  of  the  modules  within  the  system  is  sub¬ 
ject  to  error  but,  once  assembled,  is  constant  for  the 
lifetime  of  the  system.  The  width  of  the  seams  is  lim¬ 
ited  by  fiber  shaping  methods,  which  ultimately  may 
be  comparable  or  smaller  than  that  of  the  fiber  cores. 
But,  even  with  near-perfect  alignment,  there  will  be 
some  information  loss  at  the  seams  (if  only  spreading 
signals  between  conjoined  fibers),  and  the  current 
prototype  imager  is  far  from  this  ideal.  Overlapping 
the  image  boundaries  would  greatly  facilitate  stitch¬ 
ing  the  image  boundaries  but  is  not  possible  with  the 
current  design  and  would  require  greatly  increasing 
camera  size  and  complexity.  Here,  we  describe  the 
part  of  the  overall  processing  flow  (Fig.  11)  that 
follows  occlusion  removal  and  radial  vignetting 


correction.  We  separate  the  occlusion  and  radial 
vignetting  corrected  images  from  the  various  sensors 
into  their  four  Bayer  color  channels,  process  them  to 
form  four  panorama  color  channels,  and  then  demo¬ 
saic  them  into  a  color  panorama.  The  active  area  of 
the  individual  sensors  is  not  fully  illuminated  by  the 
image  being  relayed  by  the  fiber  bundles.  The  images 
from  the  different  sensors  must  be  distortion  cor¬ 
rected,  rotated,  cropped,  and  intensity  balanced  prior 
to  the  individual  image  data  alignment  needed  to 
form  a  contiguous  panorama  for  each  color  channel. 
One-dimensional  interpolation  from  each  boundary 
edge  of  valid  data  is  used  to  restore  lost  data  at 
the  image  seams.  Vignetting  effects  are  compensated 
using  a  modified  ramp  function  to  amplify  the  lower 
signals  that  appear  at  the  extreme  field  angles. 
Gamma  correction  is  then  performed  on  the  four 
color  channel  panoramas.  After  the  four  color  chan¬ 
nels  are  processed  as  described  to  form  panorama 


Fig.  12.  Schematic  showing  the  fiber-coupled  sensors  in  the  prototype  and  their  respective  field-of-view.  The  six  sensors  gather  image 
data  across  a  126°  field-of-view. 
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(b) 


Fig.  13.  Distortion  correction  and  image  stitching,  (a)  Unprocessed  sensor  data  from  two  individual  sensors  [Sensors  4  and  5  in 
Figs.  1(b)  and  12]  show  barrel  distortion  and  information  loss  at  the  sensor  adjacent  boundary,  (b)  Distortion  is  corrected,  and  the  seam 
is  interpolated  to  stitch  the  images  together.  The  ground-truth  mapping  angle  markings  are  shown  in  red  to  indicate  residual  error  in 
distortion  correction. 
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Fig.  14.  Image  taken  using  a  conventional  wide  field-of-view  camera  (Canon  5D  mark  II  with  EF  8-15  mm  F /4  fisheye  lens)  with  similar 
characteristics  as  the  fiber-coupled  monocentric  lens  imager.  Magnified  regions  show  the  image  resolution. 


color  channels,  they  are  interpolated  to  form  a  single 
white  balanced  color  panoramic  image. 

The  highly  localized  structural  and  color  artifacts 
due  to  the  fiber  bundle  limit  the  effectiveness  of  de¬ 
mosaic  interpolation  techniques  that  rely  on  texture, 
edges,  or  gradient  image  content.  After  evaluating 
several  interpolation  techniques,  we  found  that  bilin¬ 
ear  interpolation  produced  the  best  resulting  images. 
A  global  saturation  adjustment  was  used  to  compen¬ 
sate  for  the  sensor’s  color  response.  The  steps  in  the 
methodology  mentioned  here  were  selected  out  of  a 
large  class  of  techniques  and  can  be  modified  or 
replaced.  For  example,  the  stitching  process  docu¬ 
mented  in  [31]  is  effective  at  balancing  the  content 
across  seams  that  exhibit  discontinuous  content 
due  to  flawed  image  performance,  producing  a  much 
less  noticeable  image  boundary. 

5.  Experimental  Setup 

Experimental  data  were  collected  in  the  lab  with  the 
monocentric  lens  using  two  adjacent  fiber-coupled 
sensors  corresponding  to  Sensors  4  and  5  in 
Figs.  1(b)  or  12.  One  sensor  imaged  the  scene  from 
0°  on  axis  to  20°,  and  the  other  sensor  imaged  from 
20°  to  40°.  This  image  boundary  was  chosen  because 
it  well  represented  an  arbitrary  image  seam  since  it 
was  not  along  the  optical  axis.  The  planar  scene  was 
positioned  at  an  orthogonal  distance  of  1  m  from  the 
camera  along  the  optical  axis.  The  scene  was  a  flat 
printed  grid  (designed  to  be  1  m  from  the  camera) 
with  centimeter  markings  and  concentric  circles 
denoting  the  angle  deviation  from  the  optical  axis. 


Laboratory  data  (Fig.  13)  enabled  us  to  quantify 
distortion  correction  and  the  amount  of  information 
loss  at  the  image  boundary  between  closely  tiled 
adjacent  sensors,  which  corresponded  to  2  mm  in 
object  space  at  a  distance  of  1  m  or  0.2°.  From  this 
characterization,  we  can  accurately  stitch  the  scene 
together.  The  calibration  described  here  needs  to  be 
calculated  only  once  since  the  monocentric  lens 
imager  parts  are  held  fixed  by  adhesive,  with  the 
exception  of  color  balance  which  is  affected  by  the 
ambient  illumination  spectrum.  This  sufficiently 
characterizes  the  imager  so  that  significantly  im¬ 
proved  panoramas  can  be  processed  from  the  individ¬ 
ual  raw  image  sensor  data.  The  complete  systematic 
methodology  we  used  to  process  the  fiber-coupled 
monocentric  lens  image  data  is  documented  in 
Fig.  11.  Partially  and  completely  processed  images 
resulting  from  this  process  are  shown  in  Fig.  13. 
In  what  follows,  we  demonstrate  that  this  process  en¬ 
ables  the  fiber-coupled  monocentric  lens  imager  to 
produce  artifact-limited  panoramas  that  suffer  from 
less  aberrations  and  distortions  than  traditional 
wide  FOV  lenses. 

We  compared  the  performance  of  the  prototype 
against  a  commercially  available  camera  (Canon 
5D  mark  II  with  EF  8-15  mm  F / 4  fisheye  lens)  by 
taking  images  of  the  same  outdoor  scene  under  iden¬ 
tical  conditions.  The  outdoor  scene  was  imaged  using 
all  six  fiber-coupled  sensors  within  the  A/ 1.35  fiber- 
coupled  monocentric  lens  imager  to  form  a  126°  x  16° 
panorama  (see  Fig.  12).  The  prototype  has  a  126°  x 
16°  field-of-view  and  resolution  of  30  megapixels  that 
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Fig.  15.  (a)  Unprocessed  and  (b)  processed  fiber  relayed  prototype  image  data  of  a  panorama  showing  significant  image  improvement 

attributed  to  the  image  processing  methodology.  Artifacts  due  to  bubbles  in  the  optical  adhesive,  broken  fibers,  and  fiber  bundle  scratches 
are  effectively  removed  by  flat-field  calibration.  The  flat-field  calibration  images  for  Sensors  1  and  4  (from  right)  are  shown  in  Fig.  9. 
Distortions  are  corrected,  the  images  aligned,  and  missing  data  is  interpolated  in  order  to  form  the  processed  panorama.  Magnified  regions 
show  that  the  prototypes  resolution  is  superior  to  the  benchmark  camera  images  shown  in  Fig.  14. 


can  be  increased  by  incorporating  more  fiber-coupled  shown  in  Fig.  14.  This  comparison  camera  was 
sensors.  The  ground-truth  comparison  image  is  chosen,  since  it  was  designed  to  meet  similar 
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performance  specifications  without  being  limited  to  a 
small  form  factor.  Figure  15  exemplifies  the  improve¬ 
ment  our  methodology  provides  to  fiber  relayed 
imagery.  Of  particular  interest  is  the  removal  of  ar¬ 
tifacts  due  to  bubbles  (in  the  optical  adhesive  used  to 
mate  the  lens  with  the  fiber),  broken  fibers,  and 
scratches.  This  first  prototype  suffered  from  more 
of  these  effects  than  subsequent  prototypes;  however, 
we  chose  to  include  this  imagery,  since  it  best  illus¬ 
trates  the  efficacy  of  the  techniques  described.  Addi¬ 
tionally,  the  significant  vignetting  effects  at  the 
edges  of  the  wide  angle  panorama  are  corrected  for 
using  an  amplification  map.  Distortion  correction 
and  interpolation  of  missing  data  at  the  boundaries 
allows  the  formation  of  panoramic  imagery.  The 
composite  image  demonstrated  in  Fig.  15(b)  shows 
that,  by  using  fiber  bundle  image  relay  along  with 
the  techniques  described  here,  we  are  now  able  to  use 
lenses  (that  form  nonplanar  image  surfaces  and  are 
otherwise  incompatible  with  current  technology)  in 
order  to  form  high-resolution  images  with  less  distor¬ 
tion  than  that  of  the  commercial  alternative.  Magni¬ 
fied  views  (enclosed  in  red)  inset  in  Figs.  14  and  15 
show  a  portion  of  the  scene  that  is  about  50°  left  of 
center  and  150  m  downrange  for  both  comparison 
cameras.  This  comparison  shows  the  prototype  ex¬ 
hibits  less  distortion,  less  chromatic  aberration, 
and  a  large  boost  in  resolution. 

6.  Discussion  and  Conclusion 

This  paper  summarizes  the  characterization  and 
image  processing  for  imaging  systems  that  uses 
fiber  bundle  image  relay,  including  a  component- 
by-component  transfer  function  characterization  of 
the  fiber-coupled  imager.  We  found  that  the  high  spa¬ 
tial  variance  of  some  of  the  individual  components  is 
reduced  when  used  in  conjunction  with  other  compo¬ 
nents,  allowing  for  more  flexibility  when  implement¬ 
ing  image  processing  techniques.  We  experimentally 
demonstrated  how  flat-field  correction  proves  benefi¬ 
cial  for  fiber-coupled  imaging  systems  by  correcting 
for  the  response  of  the  lens,  fiber  bundle,  and  sensor, 
all  of  which  produce  fixed  pattern  artifacts.  We 
showed  how  to  stitch  together  panoramas  taking  into 
account  artifacts,  image  transfer  uniformity,  map¬ 
ping,  and  the  registration  of  individual  sensor  data 
prior  to  interpolation  of  image  information  not  cap¬ 
tured  between  adjacent  sensors.  Using  this  method¬ 
ology,  we  formed  a  30  megapixel  126°  field-of-view 
fiber-coupled  panoramic  image,  which  compares 
favorably  in  image  resolution  and  distortion  to  the 
benchmark  commercial  camera  whose  volume  (for 
lens  and  sensor  only)  is  more  than  10  times  the  fiber- 
coupled  prototype. 
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